Signal coding

ABSTRACT

The current frame of the picture is compared block-by-block with the previous frame to identify changed picture elements (pixels). The resulting matrix is matched (&#34;vector quantized&#34;) to one of a set of standard matrices (&#34;VQ shapes&#34;). Only those pixels flagged by the selected VQ shape are transmitted plus a &#34;VQ shape code&#34;. This technique is modified in that the current frame is also compared with a &#34;background&#34; frame stored at the transmitter and at the receiver. Where, for any block, all the pixels flagged by the VQ shape are indicated by the second comparison as being the same as the background, the pixels are not sent--instead a shape code plus a &#34;copy background&#34; instruction is transmitted.

This application is related to commonly assigned copending applicationSer. No. 07/364,441 filed June 2, 1989 naming Messrs. Carr, Perini,O'Donnell and Leaning as inventors and claiming priority dates of Aug.28 1987 and Oct. 19, 1987 via PCT/GB88/00709 filed Aug. 26 1988.

The present invention relates to coding of video signals, especiallyusing conditional replenishment coding, where information is generallytransmitted only in respect of elements of a frame of the picture whichhave changed relative to a previous frame; the transmitted data beingused at a receiver to update a stored version of the picture.

Thus picture elements of each block of an image to be coded are comparedwith those of the corresponding block of a previously coded image todetermine whether the block has changed between the two images; if so,picture element data are generated for output.

Such a system is described in international patent application publishedunder no. WO86/03922, which also proposes that the block be comparedwith the corresponding block of an earlier (reference) image. If theyare deemed to be the same, no picture data are generated--instead, acodeword is produced to indicate that the receiver is to obtain its datafrom a locally stored replica of the reference image.

According to one aspect of the present invention, there is provided amethod of coding an image comprising, for each of a plurality of blocksof an image:

(i) comparing picture elements of the block with those of thecorresponding block of a previously coded image to produce a matrix ofvalues each indicating whether the corresponding element is, inaccordance with a predetermined criterion, deemed to have changedbetween the two images;

(ii) matching the matrix to one of a predetermined set of such matriceseach of which identifies a region of the block as being deemed to havechanged, and generating a codeword identifying that one matrix of theset;

(iii) comparing picture elements of the block with those of thecorresponding block of a reference image composed of blocks from atleast one earlier frame to determine whether the block is, in accordancewith a predetermined criterion, deemed to have changed between the twoimages;

(iv) coding for output those elements within the identified region,unless all the elements within the region are identified by comparisonstep (iii) as being unchanged relative to the reference image, whereupona codeword indicating this is generated.

In another aspect, the invention provides an apparatus for coding animage comprising,

(i) means for comparing, for each of a plurality of blocks of an image,picture elements of the block with those of the corresponding block of apreviously coded image to produce a matrix of values each indicatingwhether the corresponding element is, in accordance with a predeterminedcriterion, deemed to have changed between the two images;

(ii) means for matching the matrix to one of a predetermined set of suchmatrices each of which identifies a region of the block as being deemedto have changed, and generate a codeword identifying that one matrix ofthe set;

(iii) a store arranged to store a reference image composed of blocksfrom at least one earlier frame and means for comparing picture elementsof the block with those of the corresponding block of the referenceimage to determine whether the block is, in accordance with apredetermined criterion, deemed to have changed between the two images;

(iv) means for coding for output those elements within the identifiedregion, unless all the elements within the region are identified by themeans for comparing as being unchanged relative to the reference image,whereupon a codeword indicating this is generated.

The matching step--often termed vector quantisation--of two-dimensionalmaps has been proposed previously for picture coding, for example inEuropean patent application Ser. No. 0239076A, where blocks of transformcoefficient blocks are classified.

The present invention, however, further exploits the vectorquantisation, in that not only can the classification be used (ifdesired) to reduce the amount of information that has to be transmittedto indicate which elements have been coded for output, but also thenumber of occasions on which a "reference" codeword is instead generatedare increased.

Some embodiments of the invention will now be described, by way ofexample, with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of a coder according to one embodiment of theinvention;

FIG. 2 shows a typical bit map produced by the change detector of thecoder of FIG. 1;

FIGS. 3a-3b illustrates a few standard bit patterns used by the vectorquantiser of the coder of FIG. 1; and

FIG. 4 is a block diagram of a decoder according to another embodimentof the present invention.

FIG. 1 shows a conditional replenishment video coder, where videosignals (assumed to be in digital form) are supplied to an input 1. Thecurrent frame of a picture is compared in a change or movement detector2 with the output of a local decoder 3. The local decoder produces a"previous frame" output which is the same as that produced by a remotedecoder; the object of the comparison is to identify those parts of thepicture which have changed so that only information concerning thoseparts needs to be sent to the decoder, to update a stored representationof the frame.

The signals are processed on a block-by-block basis--an 8×8 block isassumed--and the change detector 2 therefore produces an 8×8 bit map. Atypical map is shown in FIG. 2, where the elements of the blockcorresponding to picture elements (pixels) which have changed are shownshaded.

In practice, transmission of information concerning only the changedelements involves a significant addressing overhead and therefore it ispreferred to match the bit map to one of a limited number--typicallyforty--of standard shapes (a few are illustrated in FIG. 3). Since it ispreferable to transmit information for an unchanged pixel than to failto transmit information for a changed pixel, the shape chosen is thesmallest (i.e. with the least number of shaded elements) which has ashaded area covering the shaded elements of the bit map. This process istermed vector quantisation (VQ) and is indicated as vector quantiser 4in FIG. 1. One transmits, for the block, a VQ number identifying thechosen shape, along with information concerning pixels deemed--inaccordance with the chosen shape--to have changed. The output data arecombined and buffered in a buffer 5 prior to passage to an output 6. Asis conventional in such systems, the buffer is used to smooth variationsin the rate at which data are generated (due to the picture elementdependent coding) and interface to a regular transmitted rate and thebuffer fullness state used to control the rate of generation (e.g. byvarying the change detector thresholds (control line 7)).

The pixels deemed to have changed are coded in an encoder 8 (for examplein the manner described in our International patent applicationPCT/GB88/00709 and European patent application no. 88307981.6.

Many televised scenes--especially in a videoconference or videotelephoneenvironment--contain moving persons or objects set against a fixedbackground. Pixels observed as changed by the change detector 2 willrelate either to objects which have changed their position (or enteredthe scene) or to parts of the background uncovered by the object. In thepresent coder, a second change detector 30 is also shown, which comparesthe current frame with a reference or background frame stored in a framewith a reference or background frame stored in a frame store 31.Acquisition of the stored background frame will be discussed furtherbelow, but in FIG. 1 is assumed to have been acquired from the localdecoder 3 in response to manual operation of a switch 32 at thecommencement of a transmission; a code being transmitted to the receiverto initiate similar action at the remote decoder.

The change detector 30 produces a bit map identifying those pixels ofthe current frame which are the same as the background.

If this shows that the new image is different from the background forany of the pixels declared as changed by the VQ shape the backgroundinformation cannot be used and information concerning those pixels istransmitted along with the VQ number.

If, however the changed area as given by the VQ shape covers only pixelswhich are identified by the detector 30 as being the same as thecorresponding pixels of the stored background frame, then the VQ numberis accompanied by a reserved codeword indicating "background" and nofurther information needs to be transmitted for the block in question.

These functions are accomplished in FIG. 1 by a comparator 33 whoseoutput 34 overrides the operation of the encoder 8.

FIG. 4 shows a decoder. An input processor 40 receives the coded inputsignals. The background frame is stored in a background store 41. Innormal operation, the processor 40 uses the pixel information receivedto update via line 42 the contents of a frame store 43, using thereceived VQ numbers via line 44 to control the frame store addressing.When, however, it receives the reserved "background" codeword (via line45), it recovers the relevant pixels (identified by the VQ number) fromthe background store 41 and enters them into the frame store 43. This isillustrated schematically by a changeover switch 46. The frame store 43is read out (by output control means 47) to produce the received videoat an output 48. As in the coder, the background store 41 is loaded fromthe decoded image in store 43 when a `load background` instruction isreceived (line 49).

Note that the local decoder 3 of FIG. 1 can be of the same constructionas the decoder of FIG. 4, although, of course, in practice the localdecoder would use the background store 31 rather than contain one of itsown.

The change detectors 2,30 can in principle be any conventionally used,or may both be as described in our above-mentioned patent applications.The vector quantiser 4 can again be a known device. One possibility isthat described in our International patent application no.PCT/GB87/00816 (published no. WO88/04084 and European patent applicationno. 8627787 (published no. 0272794).

Simpler but cruder, another option is to convert the 8×8 matrix to a 4×4matrix by creating each `new` element as an OR function of four `old`elements; this reduces the number of elements in the matrix to a size(16) which can be used to address a look-up table--in the form of a 64Kbyte read only memory in which the appropriate VQ numbers are stored.

The background scene may be `frozen` manually at the commencement of atransmission. It may, but does not have to, consist solely of a fixedbackground. For example it may include seated figures (thereby coveringthe situation where a person momentarily passes his hand over his faceif the face forms part of the "background", the face does not have to beretransmitted).

It may be desirable to include provision for updating the background.For example, the incoming video could be monitored at the coder (by aunit 35 shown dotted in FIG. 1) and parts of the picture which differfrom the original background but have remained unchanged for apredetermined period of time inserted into the frame store, a signalbeing sent (via line 36) to the decoder to instruct it to do likewise.

An alternative method of updating the background store abandons anyattempt at identifying genuine background, but instead forms a referenceimage which is a composite of blocks taken from preceding images over aperiod. Each frame period, data for a few (e.g. eight) selected blocksscattered over the image area is entered into the background stores, theblock selection being such that different blocks are selected from eachframe until the whole image area has been covered. Assuming 1024 blocksper frame at 25 frames per second, this represents a period ofapproximately 40 seconds. The blocks could be loaded from the framestore 43 of the decoder, and a corresponding store in the local decoder3, the change monitor then being replaced by a simple address generator35 to select the appropriate blocks. Clearly, this is less effective, inthat some blocks of the reference image will not represent backgroundmaterial, but a significant coding advantage is still obtained, and ithas the merit of simplicity.

A variation of this approach provides that data for selected blocks,instead of being drawn from the decoded image, are actuallytransmitted--i.e. transmission of the whole block is forced even if onlya part, or none, of it is indicated by the detector 2 and quantiser 4 asmoving. In this case, the frame stores 31, 41 take their input from theinput 1 and processor 40 respectively. This also has the benefit ofensuring that transmission errors on blocks which rarely change do notpersist in the decoded image.

We claim:
 1. A method of coding an image comprising, for each of aplurality of blocks of an image:(i) comparing picture elements of theblock with those of the corresponding block of a previously coded imageto produce a matrix of values each indicating whether the correspondingelement is, in accordance with a predetermined criterion, deemed to havechanged between the two images; (ii) matching the matrix to one of apredetermined set of such matrices each of which identifies a region ofthe block as being deemed to have changed, and generating a codewordidentifying that one matrix of the set; (iii) comparing picture elementsof the block with those of the corresponding block of a reference imagecomposed of blocks from at least one earlier frame to determine whetherthe block is, in accordance with a predetermined criterion, deemed tohave changed between the two images; (iv) coding for output thoseelements within the identified region, unless all the elements withinthe region are identified by comparison step (iii) as being unchangedrelative to the reference image, whereupon a codeword indicating this isgenerated.
 2. A method of coding an image comprising, for each of aplurality of blocks of an image:(i) comparing picture elements of theblock with those of the corresponding block of a previously coded imageto produce a matrix of values each indicating whether the correspondingelement is, in accordance with a predetermined criterion, deemed to havechanged between the two images; (ii) matching the matrix to one of apredetermined set of such matrices each of which identifies a region ofthe block as being deemed to have changed, and generating a codewordidentifying that one matrix of the set; (iii) comparing picture elementsof the block with those of the corresponding block of a reference imagecomposed of blocks from at least one earlier frame to determine whetherthe block is, in accordance with a predetermined criterion, deemed tohave changed between the two images; (iv) coding for output thoseelements within the identified region, unless all the elements withinthe region are identified by comparison step (iii) as being unchangedrelative to the reference image, whereupon a codeword indicating this isgenerated; and (v) wherein in the coding of the elements within theidentified region, at least some of the elements are coded as thedifference between that element and a predicted value for that elementderived from one or more previously coded elements of the block, thesequence of coding of the elements being dependent on the orientation ofthe identified region.
 3. An apparatus for coding an imagecomprising,(i) means for comparing, for each of a plurality of blocks ofan image, picture elements of the block with those of the correspondingblock of a previously coded image to produce a matrix of values eachindicating whether the corresponding element is, in accordance with apredetermined criterion, deemed to have changed between the two images;(ii) means for matching the matrix to one of a predetermined set of suchmatrices each of which identifies a region of the block as being deemedto have changed, and generate a codeword identifying that one matrix ofthe set; a store arranged to store a reference image composed of blocksfrom at least one earlier frame and means for comparing picture elementsof the block with those of the corresponding block of the referenceimage to determine whether the block is, in accordance with apredetermined criterion, deemed to have changed between the two images;(iv) means for coding for output those elements within the identifiedregion, unless all the elements within the region are identified by saidmeans for comparing as being unchanged relative to the reference image,whereupon a codeword indicating this is generated.
 4. A decoder for usewith the coder of claim 3, comprising:(a) a frame store for storing areceived image; (b) a second frame store; (c) control means responsiveto received data to update the contents of the second frame store usingthe received data and responsive to a received codeword to update thesecond frame store with information from the first frame store; (d)means for repetitively reading the contents of the second frame store toproduce a video output signal.
 5. An apparatus for coding an imagecomprising:(i) means for comparing, for each of a plurality of blocks ofan image, picture elements of the block with those of the correspondingblock of a previously coded image to produce a matrix of values eachindicating whether the corresponding element is, in accordance with apredetermined criterion, deemed to have changed between the two images;(ii) means for matching the matrix to one of a predetermined set of suchmatrices each of which identifies a region of the block as being deemedto have changed, and generate a codeword identifying that one matrix ofthe set; (iii) a store arranged to store a reference image composed ofblocks from at least one earlier frame and means for comparing pictureelements of the block with those of the corresponding block of thereference image to determine whether the block is, in accordance with apredetermined criterion, deemed to have changed between the two images;(iv) means for coding for output those elements within the identifiedregion, unless all the elements within the region are identified by saidmeans for comparison as being unchanged relative to the reference image,whereupon a codewood in dicating this is generated; and (v) means forcomparing successive images to identify parts of the image which haveremained unchanged for a predetermined period of time, and in the eventof such parts being identified, to:(a) update the stored referenceimage; and (b) code for output data indicating which parts of thereference image have been this updated.
 6. An apparatus for coding animage comprising;(i) means for comprising, for each of a plurality ofblocks of an image, picture elements of the block with those of thecorresponding block of a previously coded image to produce a matrix ofvalues each indicating whether the corresponding element is, inaccordance with a predetermined criterion, deemed to have changedbetween the two images; (ii) means for matching the matrix to one of apredetermined set of such matrices each of which identifies a region ofthe block as being deemed to have changed, and generate a codewordidentifying that one matrix of the set; (iii) a store arranged to storea reference image composed of blocks from at least one earlier frame andmeans for comparing picture elements of the block with those of thecorresponding block of the reference image to determine whether theblock is, in accordance with a predetermined criterion, deemed to havechanged between the two images; (iv) means for coding for output thoseelements within the identified region, unless all the elements withinthe region are identified by said means for comparison as beingunchanged relative to the reference image, whereupon a codewordindicating this is generated; and (v) means arranged in operation toeffect periodic replacement of a minority of the blocks of the referenceimage stored in the store by the corresponding blocks of a recent image,the blocks being differently selected for successive replacements suchthat the store always contains a composite image made up of blocks froma plurality of frames of the image being coded.
 7. A method for codingan image comprising:(i) comparing, for each of a plurality of blocks ofan image, picture elements of the block with those of the correspondingblock of a previously coded image to produce a matrix of values eachindicating whether the corresponding element is, in accordance with apredetermined criterion, deemed to have changed between two images; (ii)matching the matrix to one of a predetermined set of such matrices eachof which identifies a region of the block as being deemed to havechanged, and generating a codeword identifying that one matrix of theset; (iii) storing a reference image composed of blocks from at leastone earlier frame and comparing picture elements of the block with thoseof the corresponding block of the reference image to determine whetherthe block is, in accordance with a predetermined criterion, deemed tohave changed between the two images; (iv) coding for output thoseelements within the identified region, unless all the elements withinthe region are identified by said means for comparison as beingunchanged relative to the reference image, whereupon a codewordindicating this is generated; and (v) comparing successive images toidentify parts of the image which have remained unchanged for apredetermined period of time, and in the event of such parts beingidentified, to:(a) update the stored reference image; and (b) code foroutput data indicating which parts of the reference image have been thusupdated.
 8. A method for coding an image comprising:(i) comparing, foreach of a plurality of blocks of an image, picture elements of the blockwith those of the corresponding block of a previously coded image toproduce a matrix of values each indicating whether the correspondingelement is, in accordance with a predetermined criterion, deemed to havechanged between the two images; (ii) matching the matrix to one of apredetermined set of such matrices each of which identifies a region ofthe block as being deemed to have changed, and generating a codewordidentifying that one matrix of the set; (iii) storing a reference imagecomposed of blocks from at least one earlier frame and comparing pictureelements of the block with those of the corresponding block of thereference image to determine whether the block is, in accordance with apredetermined criterion, deemed to have changed between the two images;(iv) coding for output those elements within the identified region,unless all the elements within the region are identified by saidcomparison in step (iii) as being unchanged relative to the referenceimage, whereupon a codeword indicating this is generated; and (v)effecting periodic replacement of a minority of the blocks of thereference image stored in the store by the corresponding blocks of arecent image, the blocks being differently selected for successivereplacements such that the store always contains a composite image madeup of blocks from a plurality of frames of the image being coded.
 9. Amethod for transmission coding each of a plurality of two-dimensionalblocks of image data within each of a series of successively occurringdigitized two-dimensional images, said method comprising:vectorquantizing detected image changes which occur within predefined portionsone of plural of each block between successively occurring image frames;comparing said detected image changes within said one predefined portionof a block to a previously stored reference image to detect whether thedetected new image data now all correspond to said reference image forthat portion and, if so, generating a corresponding codeword; and codingfor an output transmission representing each successive block either(i)the actual values for the vector quantized portion of that block; or(ii) said codeword.
 10. Apparatus for transmission coding each of aplurality of two-dimensional blocks of image data within each of aseries of successively occurring digitized two-dimensional images, saidapparatus comprising:means for vector quantizing detected image changeswhich occur within one of plural predefined portions of each blockbetween successively occurring image frames; means for comparing saiddetected image changes within said one predefined portion of a block toa previously stored reference image to detect whether the detected newimage data now all correspond to said reference image for that portionand, if so, generating a corresponding codeword; and means for codingfor transmission output representing each successive block either(i) theactual image values for the vector quantized portion of that block; or(ii) said codeword.